Hackathon X – Vol. 2

What do developers think?

From 25 to 27 June 2021, Hackathon X – Vol. 2 took place as a cooperation between Badisches Landesmuseum Karlsruhe, Allard Pierson Amsterdam and Hack & Söhne. More than 140 people from all over the world took part in the online developer event, which was held under the motto “The Digital Museum”. It dealt with challenges on the topics of games, platforms and artificial intelligence and developed creative solutions within 48 hours. Inspiring tech talks and mentoring, online yoga and social gaming on the digital platform Gather.Town accompanied the weekend. Over 20 projects were submitted by 100 people, of which 6 winning projects in 2 sections convinced an interdisciplinary jury.

Graphical user interface, text Description automatically generated

Creative Solutions with Artificial Intelligence in the Museum

In the section on “Artificial Intelligence in the Museum”, data sets from Badisches Landesmuseum and Allard Pierson were made available. The challenge was to develop concepts and applications that use artificial intelligence to make the museums’ digital content available to users in a personalised way.

The winning projects of the “AI Challenge”:

A screenshot of a computer Description automatically generated with medium confidence

All projects can be viewed here.

Development Plan

Following Hackathon X – Vol. 2, some teams were invited to continue working on their ideas and concepts in an AI development phase. Two teams took up this challenge: Nandakishor M with Image Chat and Richárd Ádám Vécsey & Axel Ország-Krisz with Pong.

Image Chat: Towards Augmented Reality & AI

As a virtual guide in the museum, the project works with an AR audio bot (Blenderbot 2.0) with internet search function and AWS to obtain information about artists or the museum. It outperforms existing models in terms of longer conversations over multiple sessions and has more knowledge and factual consistency than before, according to reviewers.

The model stores the knowledge gathered during a conversation in a long-term memory and uses this experience to have long-term conversations. During a conversation, the model searches the internet by making its own queries, reading the results and taking them into account when formulating an answer.

The project is based on the following papers:

Komeili, M., Shuster, K., & Weston, J. (2021). Internet-Augmented Dialogue Generation. https://doi.org/10.48550/arXiv.2107.07566

Xu, J., Szlam, A., & Weston, J. (2012). Beyond goldfish memory: long-term open-domain conversation. https://doi.org/10.48550/arXiv.2107.07567

About the developer:

Nandakishor M. is a Udemy Artificial Intelligence teacher and CEO of DeepFlow, a deep tech start-up from India. He is an expert in computer vision, NLP, robotics and IoT.

Laudation:

“The conversational AI-based image chat app is an ambitious and timely idea for which there is already a working prototype. Using an image as a conversation starter is a great idea, and the combination of text and image is very promising for providing additional information or recommending other museum objects that are relevant to the user.”

– Jaap Kamps, University of Amsterdam, member of the Hackathon jury

“In the Development Plan, the project has been steadily and creatively developed, incorporating new features and showing the possibilities of image recognition, voice output and the combination of augmented reality and AI. The highly creative and future-oriented development deserves the prize!”

– Sonja Thiel, xCurator project lead

Pong: Artwork Recommendation System

Pong is an AI- and ML-based artwork recommendation system that makes predictions for artworks based on different approaches.

Recommendations are part of our daily lives: From choosing music or movies, to the ideal programme and wash time of washing machines, to the autocorrect feature of messaging apps: we constantly encounter AI-, ML- or simple algorithm-based recommendation systems.

The demo version of the project recommends 27,898 artworks based on 2,074 keywords, 7 different colour categories or outputs from AI or ML models. Pong aims to develop a system that is easy to maintain and has a low cost of ownership as it does not require the usage of cloud infrastructures or services.

About the developers:

Dr Richárd Ádám Vécsey is as a freelance deep learning developer and data scientist who switched from law to IT. He enjoys playing with AI to find complicated solutions to simple problems.

Dr Axel Ország-Krisz is as a freelance deep learning developer, data scientist and career jumper from law. He believes that artificial intelligence should be accessible to everyone so that people can focus on important things, such as society and general values, instead of dealing with the problems of daily life.

Laudation:

“Pong has taken the idea of recommending artworks further in the Development Plan, and, to do so, has worked extensively with the museums’ data and metadata. The recommendation system was differentiated in terms of rating systems, and options for reaction-based rating and personality-based rating were used. After a warm-up phase, users are presented with artworks based either on colour, content (keywords) or a combination of both. The critical examination of recommendation algorithms is an important contribution to the future of museums.”

– Sonja Thiel, xCurator project lead

Project information: https://github.com/hyperrixel/pong

Resources